Flash backed DRAM module storing parameter information of the DRAM module in the flash

ABSTRACT

A device includes volatile memory; one or more non-volatile memory chips, each of which is for storing data moved from the volatile-memory; an interface for connecting to a backup power source arranged to temporarily power the volatile memory upon a loss of power from a primary power source; a controller in communication with the volatile memory and the non-volatile memory, wherein: the controller is programmed to move data from the volatile memory to the non-volatile memory chips upon a loss of power of the primary power source of the volatile memory; and parameters describing the volatile memory are stored in at least one of the non-volatile memory chips that store the data moved from the volatile memory. In some aspects the parameters include serial presence detect information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to U.S. patent application Ser. No.12/369,027, filed Feb. 11, 2009; U.S. patent application Ser. No. 12/369032, filed Feb. 11, 2009; U.S. patent application Ser. No. 12/369,040,filed Feb. 11, 2009; U.S. patent application Ser. No. 12/369 052, filedFeb. 11, 2009; U.S. patent application Ser. No. 12/369,076, filed Feb.11, 2009; U.S. patent application Ser. No. 12/369,079, filed Feb. 11,2009; and PCT Patent Application No. PCT/US09/33755, filed Feb. 11,2009.

TECHNICAL FIELD

The disclosed subject matter relates to a flash backed dual in-linememory module (DIMM) module.

BACKGROUND

Digital processing devices, such as, for example, RAID systems sometimesuse memory caches, for example, to improve performance of read and writeoperations. Caches are often implemented using volatile memory. However,if the power source of the volatile memory fails, the data stored in thevolatile memory can be lost. In addition, a volatile memory device, suchas a DRAM memory module typically requires certain parameters thatdescribe properties of DRAM devices making up the module to be placed ina separate non-volatile memory located on the memory module. One exampleof this is Serial Presence Detect (SPD). However, the storing of thisinformation can require the addition of an entire non-volatile memory tothe volatile memory just for this purpose.

SUMMARY

This disclosure relates to a flash backed dual in-line memory module(DIMM) module including a non-volatile memory, a volatile memory, and acontroller. During normal operation the DIMM is powered by a primarypower source. When the primary power source is interrupted, a backuppower source supplies sufficient temporary power to the DIMM so that thecontroller can transfer data from the volatile memory into thenon-volatile memory before power from the backup power source isdepleted. When the primary power source becomes available again, thecontroller transfers the data that was stored in the non-volatile memoryback into volatile memory.

In one aspect, a device includes volatile memory; one or morenon-volatile memory chips, each of which is for storing data moved fromthe volatile-memory; an interface for connecting to a backup powersource arranged to temporarily power the volatile memory upon a loss ofpower from a primary power source; a controller in communication withthe volatile memory and the non-volatile memory, wherein: the controlleris programmed to move data from the volatile memory to the non-volatilememory chips upon a loss of power of the primary power source of thevolatile memory; and parameters describing the volatile memory arestored in at least one of the non-volatile memory chips that store thedata moved from the volatile memory. In some aspects, the parametersinclude serial presence detect information.

In another aspect, a method includes moving data from a volatile memoryto non-volatile memory chips, each of which is for storing data movedfrom the volatile-memory, upon a loss of power of a primary power sourceof the volatile memory based on parameters describing the volatilememory stored in at least one of the non-volatile memory chips thatstore the data moved from the volatile memory while the volatile memoryis temporarily powered by a backup power source. In some aspects, theparameters include serial presence detect information.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a dual in-line memory module (DIMM).

FIG. 2 is a block diagram of a capacitor based backup power supply thatcan be used to power components of FIG. 1 in the event of a powerfailure.

FIG. 3 is block diagram of a battery based backup power supply that canbe used to power components of FIG. 1 in the event of a power failure.

FIG. 4 is a block diagram illustrating example states and transitions ofthe DIMM illustrated in FIG. 1.

FIG. 5 shows various read operation timing diagrams of anInter-Integrated Circuit (I2C) interface of the DIMM illustrated in FIG.1.

FIG. 6 shows various write operation timing diagrams of anInter-Integrated Circuit (I2C) interface of the DIMM illustrated in FIG.1.

FIG. 7 shows the architecture of the DIMM illustrated in FIG. 1.

FIG. 8 shows the architecture of a power supply used to power the DIMMillustrated in FIG. 1.

FIG. 9 is a more detailed illustration the backup power supply of FIG.2.

DETAILED DESCRIPTION

Referring to FIG. 1, the described embodiment of the invention is a dualin-line memory module (DIMM) 100 that includes volatile memory 120,non-volatile memory 130, isolation logic 140, and a controller 110. DIMM100 is connected to a primary power source (not show) to support normaloperation and is also connected to a backup power source 200 (see FIG.2). When DIMM 100 is operating under power supplied by the primary powersource, an external system (e.g. a RAID system) stores data in and readsdata from volatile memory 120 through interface 105. However, when thepower from the primary power source 200 is interrupted, a backup powersource supplies sufficient temporary power to DIMM 100 so thatcontroller 110 can cause isolation logic 140 to isolate volatile memory120 from the external system and then transfer data from volatile memory120 into non-volatile memory 130 before power from backup power source200 is depleted. When the primary power source becomes available again,controller 110 transfers the data that was stored in non-volatile memory130 back into volatile memory 120 and causes isolation logic 140 toreconnect volatile memory 120 to the external system.

Volatile memory 120 is a DRAM array that includes various DRAM chips,e.g., DRAM chips 121 and 122. Non-volatile memory 130 includes variousflash memory devices, e.g., flash devices 131 and 132. Due toconstraints of DIMM 100, all the data stored in volatile memory 120cannot be moved to non-volatile memory 130 at one time. One of theseconstraints is that the flash devices of non-volatile memory 130 cannotbe written into as fast as the DRAM devices of volatile memory 120 canbe read from. To account for this discrepancy, data is moved fromvolatile memory 120 to non-volatile memory 130 one DRAM chip at a time.In addition, during the transfer of data from volatile memory 120 tonon-volatile memory 130, DRAM chips not being actively transferred areput into a low power state that maintains the data stored in them butconsumes less power than a normal operating state. In the DRAM chips ofvolatile memory 120, this low power state is self-refresh mode. Byputting the DRAM chips that are not being actively transferred into alow power state, module 100 requires less power during the backupoperation than it would otherwise. This allows, for example, for asmaller and less expensive backup power source to be used.

FIG. 2 shows a block diagram of such a power source. Specifically, FIG.2 shows a electrochemical double layer (EDL) capacitor backup supplymodule 200 that interacts with DIMM 100 using interface lines (e.g.,power, I/O, etc.) 170 (EDL capacitors are also know as super capacitorsand ultra capacitors). Capacitor 210 supplies the backup power to DIMM100 upon a power failure of the primary power source of DIMM 100.Charger and monitor 220 charge capacitor 210 and perform state-of-healthmonitoring of capacitor 210 so that, for example, DIMM 100 can bealerted if capacitor 210 is failing and can no longer provide backuppower. In some cases, a battery may be selected instead of a capacitor.For example, FIG. 3 shows a block diagram of a battery backup powersupply module 300 that can be used instead of module 200.

Non-volatile memory 130 is embedded with the serial presence detect(SPD) information for volatile memory 120 (e.g., information thatdescribes the size and speed of DRAM chips in volatile memory 120). Byusing non-volatile memory 130 to store the SPD information of volatilememory 120, volatile memory 120 avoids the need for having a separateEEPROM module for storing this information. Avoiding the addition of aseparate EEPROM saves costs, reduces the size module 100, and reducesthe number of components required.

DIMM 100 includes two I2C buses between the external system andcontroller 110. I2C busses are typically used to attach low-speedperipherals to various devices when, for example, simplicity and lowmanufacturing cost are more important than speed. The first I2C bus isfor accessing the Serial Presence Detect (SPD) EEPROM (the “SPD I2Cbus”). This is defined by the standard JEDEC spec. The second I2C bus isused to access other module 100 information, such as, status informationand state-of-health (SoH) information for controller 110, non-volatilememory 130, and backup power source 200 (the “NVDIMM I2C bus”). Thestatus information can include, for example, current state of the flashmemory (written, erased, erasing, defective, etc.); number of bad blocksswapped out; number of spare blocks remaining; total number of downloadcycles completed; number of ECC errors in last download; number of ECCerrors in last restore; status of last download (in progress, completedno errors, completed with errors, etc.); status of last restore (inprogress, completed no errors, completed with errors, etc); flash headerinformation. The SoH information can include, for example, current stateof the backup power source (charged, discharged, charging, etc.),whether any capacitors making up a backup power source have failed (andif so, which capacitors have failed), and type of backup power source(e.g., capacitor or battery).

Block Diagram Details

Flash memory 130 provides the nonvolatile storage on the DIMM and isimplemented using Secure Digital (SD)/MultiMediaCard+ (MMC+). Controller110 can support various arrangements, for example, four independentSD/MMC+ interfaces to four SD mass storage devices each operating with20 Mbyte/sec bandwidth using a 4-bit data bus or four MMC+ mass storagedevices each operation with 40 Mbyte/sec bandwidth using an 8-bit databus. One advantage of using SD/MMC technology is the complexity ofmanaging flash memory is hidden from controller 110 using a simple, lowpin count interface. The flash memories can be implemented in a singledevice, for example, the SanDisk iNAND, or can be constructed using adiscrete SD controller with separate NAND memory devices on the sameDIMM. In either case, the SD/MMC controller is responsible for ECC andbad block management according to the NAND technology used.

Serial Presence Detect (SPD) data is stored in the first 256 bytes ofthe flash memory attached to the first SD/MMC+ interface (i.e., flashchip 131). The typical write protection mechanism is implemented usingflags stored within the flash configuration space. Controller 110implements a read-cache, write-through mechanism for the SPD data, wherethe SPD information can be stored in a cache on controller 110 (inaddition to on a flash chip 131). During system power up, controller 110fetches the SPD data from the flash memory. Read operations on the SPDI2C interface use the cached data while write operations are immediatelywritten to the flash memory. During the write operation to the flashmemory, the SPD I2C interface will ignore any read or write requests.

The status information data is stored in the second 256 bytes of theflash memory attached to the first SD/MMC+ interface. This interfaceallows the user to monitor and configure the operation of thenon-volatile functions. The region is also used to track the systemstate during the last power cycle. Controller 110 implements aread-cache, write-cache mechanism for the configuration data, where thestatus information can be stored in a cache on controller 110 (inaddition to on a flash chip 131). During system power up, the FPGAfetches the data from the flash memory. Read and write operations on theNVDIMM I2C interface use the cache data. Cache data is written backduring power-off and power-loss (backup) events.

Controller 110 is an advanced embedded processor with a custom 133 MHzDDR controller, four custom SD/MMC+ host interfaces, the SPD I2Cinterface, and the NVDIMM I2C interface. The microprocessor can be, forexample, a soft 32-bit Altera NIOS RISC processor executing firmwarefrom the internal memory instance in the FPGA (programmable read-onlymemory (PROM) 115). The processor controls the operating state of module100 data movement between the DDR and SD/MMC+ interfaces andcommunication on the SPD and NVDIMM I2C interfaces. The custom DDRinterface allows controller 110 to manipulate the DRAM array on a perbyte-lane basis. The interface has individual control of the CKE signalsallowing each device in the DRAM array to be controlled. The controlleruses the first 8 bytes in each byte lane in the array to set theinternal phase alignment of the bus. The four custom SD/MMC+ interfacesare designed for embedded applications where features such as hot plugare not required. The interface supports 1-bit, 4-bit and 8-bitoperation at clock speeds up to 50 MHz. The interfaces also can operatetogether synchronizing four SD/MMC+ cards allowing high-bandwidth readand write operations without large amounts of data buffering. Forapplications requiring the SD/MMC+ cards to be removed, the FPGA hostinterface allows the cards to be reordered for situations where thecards are not installed in the correct order.

Volatile Memory 120 is a DRAM array. Various examples configurationsincluding 8 bits of error correcting code (ECC) for every 64 bits ofactual data are shown in the table below. In the example with two Gigabytes of NVDIMM, one rank can be turned on and off depending on currentmemory requirements. Turning off a rank when it is unneeded saves power.When data (actual data and ECC) is moved from volatile memory 120 tonon-volatile memory 130, non-volatile memory 130 stores the actual dataand ECC without a distinction between the two stored in the non-volatilememory 130. When the data is moved back from non-volatile memory 130 tovolatile memory 120, controller 110 restores the actual data and ECCback into volatile memory 120 as is appropriate for the particular DRAMdevices being used.

NVDIMM DRAM # of Total Device Configuration DRAMs Ranks 256 Mbyte 512Mbit  32 Mwords × 16 bits 5 1 512 Mbyte  1 Gbit  64 Mwords × 16 bits 5 1 1 Gbyte  1 Gbit 128 Mwords × 8 bits 9 1  2 Gbyte  1 Gbit 128 Mwords × 8bits 18 2

PLL 161 is a high performance, low skew, PLL-based, zero-delay bufferthat distributes a differential input clock signal to the DRAM array.The DDR clock from the edge connector is multiplexed with the DDR2 clockfrom controller 110 to prevent PLL 161 from entering into its low powerstate and tristating its outputs. In this example design, the selectedPLL must operate at the desired system rate as well as a slower DDRcontroller rate. In general, PLLs bypass themselves and operate as asmall-delay buffer at the slowest clock rates

Control and address signals are re-driven through registers 162 to theDRAM devices on the following rising clock edge (data access is delayedby one clock). Controller 110 uses tristates to access the address andcontrol signals. When controller 110 controls DRAM 120, the register isisolated from the edge connector using FET bus switches 163 andcontroller 110 can directly drive the register inputs. When the systemcontrols the DRAM array, the FET bus switches 163 are on and the FPGAtristates its outputs. The CKE signals are treated differently from theother control signals. The switching between the two operational modesis glitchless to ensure DRAM 120 remains in self-refresh mode. For thesesignals, FET switches 163 are used to multiplex between the edgeconnector (leading to the system) and controller 1 10.

The high-bandwidth FET mulitplexers 163 and 164 are designed to supporthigh-bandwidth applications such as memory interleaving, bus isolationand low-distortion signal gating. The FET multiplexers 163 and 164isolate module 100 from the system bus during a power-loss event. TheFET multiplexers use a charge pump to elevate the gate voltage of thepass transistor, to provide a low and flat on-state resistance. The lowand flat on-state resistance allows for minimal propagation delay andsupports rail-to-rail switching on the data input/output (I/O) ports.The FET multiplexers also feature low data I/O capacitance to minimizecapacitive loading and signal distortion on the data bus.

Depending on the configuration of module 100, not all data stored involatile memory 120 is backed up to non-volatile memory 130. Instead,module 100 can be configured to backup (and later restore) data storedin select portions of volatile memory 120. Information stored innon-volatile memory is typically key/directory information used todetermine the location of information (e.g., files) in a file system.Key/directory information is critical information that essentially allusers will choose to backup. However, other types of information canalso be stored in volatile memory 120. For example, software programinformation that does not change (e.g., a “.exe” file) can be stored involatile memory 120. Controller 110 includes registers that allow a userto segment volatile memory 120. A starting address is stored in oneregister and an ending address is stored in a second register. All datastored between these two addresses will be backuped and restored. Datastored outside of these addresses will not be backuped/restored. Thevalues of these registers are controlled through the NVDIMM I2C bus.Users, may choose for example, specify the starting and ending addressessuch that only key/directory information is backedup/restored. Onereason for choosing to only restore key/directory information is toimprove restore time by not wasting time restoring information that doesnot need to be restored from non-volatile memory 130 (e.g., a “.exe” hasvery likely not changed and can be loaded from the host system whenrequired).

Module 100 can be configured to support various numbers of flash chips(e.g., 1-4) and is its firmware is programmed depending on the selectednumber. The number of flash chips used can be based on, for example, thesize of volatile memory that needs to be backed up and the time in whichthe backup must occur (e.g., the amount of time backup power can besupplied) or on reaching a desired restore speed (i.e., more flashdevices allow for a quicker restore time). For example, for a controllerthat can support up to four flash chips, the controller would have fourbusses. Each of the busses can be connected (or not connected) to aflash chip depending on the selected number of flash chips. The selectednumber of flash chips (e.g., 1, 2, or 4) are connected to the busses andsoldered onto a printed circuit board (PCB). For a module 100 that isdesigned to accommodate up to four flash chips, if only two flash chipswere installed, the remaining space for the not-installed two flashchips remains empty and controller 110 is programmed to only attempt tocommunicate with the two installed flash chips. For a constant backuptime or restore time, the number of flash chips can be increased inproportion with the size of the volatile memory. Alternatively, thebackup time and restore time can be reduced by increasing the number offlash chips.

Signal Descriptions

Module 100 implements a 72 bit DDR2 memory interface with a 244 pinmini- DIMM connector. The connector signal assignments are defined inJEDEC Standard 21C Page 4.20.14-2, DDR2 Registered Mini-DIMM DesignSpecification (currently available from www.jdec.org). Signalscorresponding to each of the 244 pins are not illustrated in FIG. 1 toavoid making FIG. 1 unreadable.

The NVDIMM_RESET signal initializes controller 110 and forces thecontroller to restart its state machine. Controller 110 is also resetwhen the standard RESET_IN input is asserted (along with, for example,volatile memory 120 and register 162). When the controller 110 is heldin reset by NVDIMM_RESET, module 100 operates normally. That is, thewhen NVDIMM_RESET is asserted, FET switches 163 are held on thusallowing the system to access the DRAM memory 120 without furtherinteraction.

The NVDIMM_PG signal reports the state of the power in the user system.When the signal is high, the system power rails are operating withinspecification. When the signal goes low, power loss is imminent andcontroller 110 moves data to flash memory 130. The system puts all DRAMdevices (e.g., 121-122) into self-refresh operation before deassertingNVDIMM_PG if the DRAM device data is to be moved to flash memory (asindicated by the NVCACHE_ENABLE signal). If NVCACHE_ENABLE is low whenNVDIMM_PG deasserts, then the data in the DRAM devices is ignored duringthe power loss event.

The NVCACHE_ENABLE signal reports the existence of cache data in theDRAM devices that should be moved to flash memory if the system powerfails. If NVCACHE_ENABLE is high when NVDIMM_PG deasserts, controller110 moves the data in the DRAM devices to flash memory. IfNVCACHE_ENABLE is low when NVDIMM_PG deasserts, the DRAM contents areignored and are not stored in flash memory 130. That last sequence usedby the system to shut down normally (e.g., without a power failure inresponse to a user requesting a shut down). When NVDIMM_PG is low,NVCACHE_ENABLE is ignored by controller 110 to prevent spurioustransitions on the signal from affecting any backup events.

During restore operations, the NVCACHE_ENABLE is used by the system tosignal to controller 110 that flash memory 130 may be erased. A dirtytag within the flash memory is not cleared until a handshake withNVCACHE_ENABLE is completed. This allows, for example, the system tohandle another power loss event during the restore operation.

After the data is moved from flash memory 130 to DRAM 120, theDRAM_AVAILABLE signal is asserted indicating the system may access thedata. When the system decides flash memory 130 should be purged (e.g.,to prevent data to be restored again after a power loss event), thesystem deasserts (falling edge) NVCACHE_ENABLE to reset the flashmemory. The system waits for NVDIMM_READY to assert before assertingNVCACHE_ENABLE again. The system can continue to use module 100 beforeNVDIMM_READY asserts, but the data will not be backed up during apower-loss event.

The CACHE_DIRTY signal indicates that flash memory 130 contains a dataimage of DRAM 120. During the BACKUP state, the CACHE_DIRTY signalindicates the start of the backup process. During POWER UP state, theCACHE_DIRTY indicates that flash memory 130 contains a backup image. Thesignal is held high until the NVCACHE_ENABLE signal is deasserted(falling edge), indicating the cache data has been read from DRAM 120.

The DRAM_AVAILABLE signal indicates when the system can access DRAM 120.When DRAM_AVAILABLE is low, controller 110 has control of DRAM 120. Whenthe signal is high, the system can take the DRAM devices (e.g., 121-122)out of self-refresh and access the data. In the event of a power up withdata in flash memory 130, DRAM_AVAILABLE will remain deasserted untilthe flash data is moved to DRAM 120. Once the signal asserts, the systemmay read and write to DRAM 120, but cannot assert NVCACHE_ENABLE untilmodule 100 is ready. A delay between the assertion of DRAM_AVAILABLE andNVCACHE_ENABLE may arise, for example, after a restore operation becausenon-volatile memory 130 is being erased or the backup power source isbeing recharged. The system may choose to only read from (as opposed toreading from and writing to) volatile memory 120 during this time.

The NVDIMM_READY signal indicates that module 100 is capable of handlinga power-loss event. The signal does not assert until the external powersource is in good health and fully charged. When configured to fullyerase flash memory 130, the NVDIMM_READY signal will also not assertuntil flash memory 130 is fully initialized to a known state. Thisfeature allows the design to support flash memory devices that cannotsupport full-speed burst write operations without erasing the flashmemory. During normal system operation (idle state), the system cannotassert NVCACHE_ENABLE until NVDIMM_READY is asserted. During backupoperation, NVDIMM_READY is deasserted. During restore operation,NDIMM_READY is deasserted. If controller 110 determines at any time thata power loss event cannot be handled correctly, for example, if the EDLcapacitor bank failed a self-test operation, controller 110 deassertsthe NVDIMM_READY to notify the system to move any cache data from theDIMM memory (e.g., to move the data to permanent storage such as a harddrive of the system).

The NVDIMM_SEATED is a pull-up pin on the DIMM pin out that allows thesystem to detect module 100. The system also can also detect module 100by attempting to read from the NVDIMM 12C interface to see if the I2Cslave responds.

The NVDIMM I2C slave interface on controller 110 provides a full-featureuser interface to controller 1 10. A user can configure and controlcontroller 110 as well as access detailed status information using theNVDIMM_SDA and NVDIMM_SCL (signals 152).

V3P3_AUX is the auxiliary 3.3V voltage rail that supplies power to thenonvolatile logic during normal system operation. During a power losscondition, module 100 switches from this supply and operates from VBACK171 (the voltage rail that is the power supply used during the backupoperation) until controller 110 turns itself off.

Module 100 also includes a third I2C interface that is located betweencontroller 110 and backup power source 200 (the “backup power supply I2Cinterface”). The backup power supply I2C interface allows controller 110to communicate with the external backup power supply module usingVBACK_SDA and VABACK_SCL. Through the interface, controller 110 candetermine the type of backup power method (e.g., EDL capacitor orbattery) as well as determine the state-of-charge and state-of-healthfor the power supply. Information communicated across the backup powersupply I2C interface can be communicated across the NVDIMM I2C interfaceas part of the SoH information. The backup power supply reset(VBACK_RESET) allows controller 110 to reset the external backup powersupply module. The VCHRG voltage rail supplies power to the EDLcapacitor charge or external battery backup power supply module. Thevoltage rail is nominal 12 volts capable of sourcing 500 mA.

Signals TEST_RX and TEST_TX make up a production test interface that isa 57.6 Kbaud serial link. During normal system operation, the testsignals are tristated and floating.

States and State Transitions

FIG. 4 illustrates a state diagram that shows various states and statetransitions of module 100. For example, module 100 is initialized to thePOWER UP state by the system reset no matter what state the module isoperating in. In this case, the module initializes all logic and fetchesconfiguration from the flash memory before determining what happened onthe last power cycle. For example, the module loads firmware from FPGAPROM 115; CACHE_DIRTY is asserted, and NVDIMM_READY and DRAM_AVAILABLEare deasserted. The SD/MMC+ flash memories are initialized to theSD/MMC+ transfer state. The flash configuration block is read todetermine the state of the last power cycle. If the DIRTY tag is set andthe previous backup operation completed successfully, the statetransitions to the WIPE state (if configured to wipe run-time area). Ifnot configured to wipe run-time area, the state transitions to theRESTORE state. If the DIRTY tag is set and the backup operation did notcomplete successfully, the state moves to the ERASE state (if the ERASEbit is set) or to the IDLE state with DRAM_AVAILABLE and CACHE_DIRTYset. If the previous ERASE state did not finish cleanly and the ERASEbit is set, then transition to the ERASE state to redo the erase cycle.If the DIRTY tag is not set, then CACHE_DIRTY is deasserted,DRAM_AVAILABLE is asserted, and the state transitions to the IDLE state.

The IDLE state is the normal operating state when the system power isapplied. If CACHE_DIRTY is already set and NVCACHE_ENABLE is thenasserted, CACHE_DIRTY is deasserted. CACHE_DIRTY can be used to confirmthe response of an unsuccessful restore operation due to an invalidbackup. If NVCACHE_ENABLE is asserted and NVDIMM_READY is asserted,asserting CACHE_DIRTY acknowledges that the controller is now operatingin a nonvolatile state (power loss will trigger a backup operation). IfNVCACHE_ENABLE deasserts, deassert CACHE_DIRTY to acknowledge that thecontroller is now operating in volatile state (power loss will not abackup operation). If the backup power source is within voltagespecification, asserting NVDIMM_READY indicates the system can support apower failure. If the backup power source fails self-test (or for anyother reason controller 110 cannot complete the backup operation),deasserting NVDIMM_READY signals the system to empty the cache. IfCACHE_DIRTY is asserted and NVDIMM_PG deasserts, power has been lost andDRAM memories contain data to be written to the flash memories. In sucha case, controller 110 deasserts DRAM_AVAILABLE and NVDIMM_READY andtransitions to the BACKUP state. If CACHE_DIRTY is deasserted andNVDIMM_PG deasserts, the DRAM memories do not contain valid data and thepower is turning off normally. In such a case, deassert DRAM_AVAILABLEand NVDIMM_READY (if set) and transition to the POWER DOWN state.

The BACKUP state is responsible for moving data from the DRAM 120 to theflash memory 130, while operating on backup power. If the GLITCH bit isnot set, controller 110 waits for CKE to go low to ensure the DRAMmemories are placed in self-refresh (the GLITCH bit is set if a backupoperation started, but during the operation the power came back up). Theon-board regulators switch to source power from the backup power. TheDDR bus is disconnected and controller 110 drives DRAM 120. Controller110 masks off the NVCACHE_ENABLE signal as the system may be poweredoff. Controller 110 asserts the DIRTY register and writes the flashconfiguration page to record the start of the backup process. If acheckpoint exists due to transitioning back from the GLITCH state,controller 110 restarts the backup from the checkpoint. Otherwise, foreach DRAM device of DRAM 120, take the DRAM device out of self-refreshand write the contents the flash memory 130. If NDIMM_PG asserts duringthe memory copy operation, put the active DRAM device back intoself-refresh, checkpoint the current backup point and move to the GLITCHstate. When all DRAM devices have been copied to flash and the flash hascompleted its programming cycle, controller 110 writes the flashconfiguration with current state information and waits for theprogramming cycle to complete. Finally, controller 110 moves module 110to the POWER DOWN state.

The POWER DOWN state handles the power down operation to prevent memorymodule 100 from restarting prematurely if system power is stillavailable. That is, NVDIMM_PG may indicate a power-loss event, butsystem power may not have been removed from the memory module.Controller 110 switches on-board regulators back to normal power ifoperating on backup power. Controller 110 tristates the interfacebetween the controller and the multiplexers 163 and connect the DRAMdevices to the DDR bus. If NVDIMM_PG ever asserts, system power is stillpresent so the system is restarted by moving to the POWER UP state.

The glitch state signifies a case in which the system temporarily lostpower, but power has returned before the backup operation completed thusallowing the user to retrieve the DRAM contents without using the flashdata. The GLITCH state may be entered multiple times during a backupoperation. There are two outcomes when in the GLITCH state. Either thepower-loss event continues and the data within the DRAM 120 is backed upto flash 130 or the data within DRAM 120 is retrieved and NVCACHE_ENABLEis deasserted. If the power loss is temporary, the memory module muststill recharge the EDL capacitor to a known state before assertingNVDIMM_READY and allow the system to rely on the nonvolatile function.To do so, module 100 records the event by asserting the GLITCH registerindicating the power restored before loss of backup power; switches theon-board regulators to source power from the normal system power;tristates controller 110 and reconnect the DDR bus to the DRAM memories;asserts DRAM_AVAILABLE indicating data is available; unmasks theNVCACHE_ENABLE signal because the system can empty the DRAM devices ofdata; if NVDIMM_PG is asserted and NVCACHE_ENABLE is deasserted, thepartial copy in the flash memories is not required, deassertsCACHE_DIRTY and transitions to the ERASE state; if NVCACHE_ENABLE isasserted and NVDIMM_PG deasserts, power has been lost and DRAM memoriescontain data to be written to the flash memories, deassertsDRAM_AVAILABLE and transitions to the BACKUP state to resume from thecheckpoint.

The wipe state, is for applications where part of the DRAM 120 is usedfor run-time, non-volatile purposes (e.g., if a user has decided tostore non-key/directory information in volatile memory 120 and haschosen not to backup/restore the non-key/directory information) In thewipe state controller 110 can be configured to zero out a singlecontinuous region in the address space. This feature prevents spuriousECC errors within the DRAM memories. Module 100: disconnects the DDR busand has controller 110 drive the DRAM memories; if the wipe function isinterrupted by loss of power indicated by NVDIMM_PG deasserting,transitions to the POWER DOWN state; for each DRAM, initializes theconfigured address space to zero and initializes the associated ECCvalues; when all DRAMs are completed, transitions to the RESTORE state.

The RESTORE state transfers flash memory 130 contents back to DRAM 120.The state is not exited until the system indicates the restored data inthe DRAMs has been read, in order to handle power-loss events duringthis state. A power-loss event during the RESTORE state causes the sameimage to be restored to the DRAM memory on the next power-on event.Module 100: If not already done so, disconnects the DDR bus and hascontroller 110 drive the DRAM memories; records in the flashconfiguration memory that the restore operation has started allowing thesystem to detect a multiple restore event; for each DRAM, copies thecontents from the flash memory to the DRAM; when all DRAMs arecompleted, updates the MODE bytes with the value stored in the flashconfiguration (system MODE value as the MODE value is read only) andputs that DRAM into self-refresh operation; if the restore function isinterrupted by loss of power indicated by NVDIMM_PG deasserting,transitions to the POWER DOWN state; tristates controller 110 andreconnects the DDR bus to DRAM 120; asserts DRAM_AVAILABLE; and whenNVCACHE_ENABLE transitions from high to low (falling edge), transitionsto the ERASE state to erase the flash contents.

Some flash memory devices require the memory to be erased in order toachieve maximum bandwidth performance during large sequential writeoperations. In such embodiments, the system at least clears any flagsindicating that a partial copy or image exists in the flash memory. Theerase operation cannot occur until the system has indicated that anydata in DRAM 120 (that is, the restored data or partially backup data)has be read from DRAM 120. Module 100 enters the ERASE state and: if anerase cycle was interrupted by another power loss, restarts the erasecycle at the beginning; tristate controller 110 and reconnect the DDRbus to the DRAM 120; assert DRAM_AVAILABLE; if required, writes theflash configuration to clear any flags indicating a potential backup ora backup image within the flash memory and to mark that an erase cyclehas started; if the ERASE register is asserted, erases the flash memorymodules; if the erase function is interrupted by loss of power indicatedby NVDIMM_PG deasserting, transitions to the POWER DOWN state; updatesthe flash configuration when the erase cycle completes normally; whenerase operation is completed, transitions to IDLE state.

I2C Interfaces

The FPGA controller has two separate slave I2C interfaces that arecontrolled using the same protocol as the industry standard two-wire I2Cserial EEPROM (i.e., the SPD I2C interface and the NVDIMM I2Cinterface). The base address for the SPD I2C interface is set at thestandard 0xA0 while the base address for the NVDIMM I2C interface is setat 0xB0. The 1-bit address offset is applied to both base addresses toallow the two interfaces to connected together if required andinter-operate with other modules. That is, SPD I2C interface and theNVDIMM I2C interface can be implemented as a single physical interfacethat use different address ranges. Controller 110 uses the first 512byte block in the flash memory attached to the first SD/MMC+ interface.The first half of the block is used for SPD data while the second halfof the block is used for flash configuration. FIG. 5 shows thefunctional operation of a read using the I2C interfaces. FIG. 6 showsthe functional operation of a write using the I2C interfaces.

Debug, Maintenance Test, and Scan

Module 100 also includes functionality for debugging, maintenance,testing, and scanning. For example, if controller is implemented usingan Altera processor, the Altera JTAG UART interface is a fully featureddebug and monitoring interface that allows the user to access thefirmware functions. Using the Altera tool suite, the firmware can bemonitored and/or overwritten with new firmware for investigationpurposes. Using the built-in UART function, the JTAG interface can beused to emulate a serial interface allowing slow-speed customcommunication. As well, for larger FPGA configurations, the firmware canincorporate a test user interface to perform diagnostic tests on theDRAM and flash memories for testing purposes. The test user interface isalso accessed through UART built into the Altera JTAG interface.

During production test, controller 110 can be held in reset to allowDRAM 120 and SD/MMC+ flash 130 to be tested. Thus, FET switches 163 and164 on the DDR interface are held in the proper state when controller110 is held in reset. At the same time, controller 110 tristates theSD/MMC+ interfaces to allow for bed-of-nail testing of the flashmemories.

Controller 110 has a production test control that configures thefirmware to run a production test on the DDR interface and SD/MMC+interfaces to ensure proper connectivity. Controller 110 reads andwrites to the DDR memory devices to exercise the address, data andcontrol signals of the DDR. As well, controller 1 10 passes sufficientdata over the SD/MMC+ command and data buses to exercise all signals. Asan end-customer production test, the NVDIMM I2C interface provides amechanism to control the backup and restore operations and directlyaccess the flash and DDR memory.

FPGA Architecture

FIG. 7 shows an example architecture of controller 110 that uses theembedded NIOS processor with the Avalon bus to connect the IP blockstogether (For example, if controller is implemented using an Alteraprocessor, the Altera JTAG). Changes to a design from Altera includeanother I2C interface (the NVDIMM I2C interface), updated GPIO andmodified firmware for the new features.

Power Supply Description

The power supply of module 100 performs a number of system operations.It generates the voltage rails required for controller 110 and itisolates module 110 from the system rails during a power failure. FIG. 8shows the structure of the power supply.

During normal operation, transistor Q3 and the diode D1 directs thesystem power VDD and V3P3_AUX to the devices on module 100. V3P3_AUX isan additional edge connector pin that supplies power for thenon-standard devices on module 100. VDD is the standard module powersource involving a number of edge connector pins. VMEM is the supplyrail powering the module 100 devices. The V3P3_AUX voltage rail isconverted to three additional supply rails required by controller 110.In general, the V1P2 and the V1P8 voltage rails supply the bulk of thecontroller power with the V2P5 rail supplying power for the FPGA analogPLLs and the V3P3 supplying power for the FPGA digital I/O, oscillatorand the SD/MMC+ cards.

Controller 110 is informed of an imminent power loss either through edgeconnector signals or through the NVDIMM I2C interface. Once this occurs,controller 110 turns on Q1 and Q4 and turns off transistor Q3 to sourcepower from the EDL capacitor bank connected to VBACK 171 and isolatingthe module from the system power. This power switch is hit-less as, inthis case for example, the power supply regulators U1, U2, U3 and U4 areconfigured not to cause controller 110 to reset or the DRAM 120 to losedata. One method of switching between power supply sources is to usediode switching. The diode D1 prevents, for example, the EDL capacitorvoltage from feeding back into the V3P3_AUX supply which may cause thesupply to glitch if a transistor is used due to charge sharing between“decoupling” capacitors on the two rails (with one of the capacitorsbeing the EDL capacitor bank). Depending on the type of power lossevent, controller 110 may be requested to stop using the EDL capacitorbank and to move back to the system power. Again, this power switch ishit-less as to prevent data loss or glitches on the power rails.

Module 100 also charges and monitors the EDL capacitor bank using theVCHRG supply. This voltage rail is specified to be used for chargingpurposes only and the module continues to operate normally even whenVCHRG is not connected. To improve the power efficiency of the EDLcapacitor bank, the minimum input voltage of U3 and U4 can be as low aspossible. This minimum input specification limits the low end of the EDLcapacitor discharge curve as the backup operation must be completebefore U3 or U4 reach their specification limit. For this reason, theV3P3 regulator U1 is a step-up regulator from the V1P8 supply allowingthe minimum EDL discharge level to be as low as possible.

While cascading regulators affects the efficiency of the power supply,the V3P3 and V2P5 are relatively low power compared to the V1P8 supply.The power loss in cascading the V3P3 regulator U1 is significantly lessthan the improved EDL capacitor bank efficiency due to the lower minimumdischarge level. A 1V decrease in the minimum EDL capacitor bank voltageis a 10% improvement in total system power while cascading V3P3regulator U1 represents a rough 30% drop in efficiency of just the V3P3supply rail (which is roughly 5 to 10% of the total system power).

The system of FIG. 1 supports four configurations of DRAM devices withdifferent numbers of SD/MMC+ memory devices. Each configuration hasdifferent PCB layouts which allow the power supply design to adjusted tosupport the different loads. The power supply design can handle 70° C.ambient (PCB) temperature with no forced air flow. Components for thepower supply design are located on one side of the PCB with a maximumheight of 4 mm. The power supply takes no more than 3 inches by 1 inchof board space including any heat spreaders.

The VDD system rail is over-constrained so that the voltage drops acrossthe transistor Q3 does not cause the VMEM supply to fall outside thedownstream device ranges. The voltage drop budget the Q3 transistorvoltage drop has been selected at arbitrary percentage of nominal value,but better performance is desirable.

The regulators shown in the diagram are functional and can be combinedinto multiple-output regulator devices. All voltage rails have monitors(the “PG” signals) which can be combined with the regulators or in aseparate device. The V3P3 monitor is separate because it monitors theV3P3 rail while the V3P3_AUX rail is the power source. The V3P3regulator does not function if VCHRG is not supplied, but this is not anerror as module 100 must still continue to operate (see FIGS. 2 and 3).To increase the energy storage efficiency of the EDL capacitor bank, theminimum allowed input voltage to the regulators is as low as possible.The table below provides additional information for the regulators.

Regulator Min Input Max Input Notes U1 V3P3 1.8 V VBACK 1, 2, 3 U2 V2P52.8 V VBACK 3 U3 V1P8 2.8 V VBACK 3 U4 V1P2 2.8 V VBACK 3 Notes: 1.Closest headroom of all the regulators and limits the performance of theEDL capacitor bank. 2. Low-quiescent current requirement as theregulator idles (capacitance load) in normal operation. 3.Specifications based on configuration. See the section on voltage railsfor specifications. Devices close to current specifications should beinvestigated as the power consumptions are still estimates.

The power transistors are responsible for moving the DRAM devices to thebackup power and isolating the backup power from the system power VDDand V3P3_AUX. The capacitor charger U6 is handles the loss-of-power andprevents the EDL capacitor bank from discharging back through thecharger. Transistors Q3 and Q4 is an n-channel MOSFETs and is controlledby the FPGA using 3.3V control signals removing the need for high-sidedrivers. Transistor Q1 is a p-channel MOSFET directly controlled by theFPGA or n-channel MOSFETs with some high-side drive mechanism (the VCHRGsupply or VCAP supplies normally cannot be used unless the modulecontinues to operate without these supplies). The currents listed in thetable below have some over-design margin so transistors close to meetingthe specification can also be used in this example design. During apower loss event, the transistors only operate until the EDL capacitorbank is discharged (e.g., a maximum of about 2 minutes).

Transistor Max VDS Max IDD On Resistance Notes Q1  12 V 2700 mA 0.020ohm 1 Q3 1.8 V 4000 mA 0.009 ohm 2 Q4 1.8   650 mA 0.056 ohm 2 Notes: 1.Based on low supply specification of VCAP (2.8 V) and 2% margin insupply voltage. 2. Based on low supply current specification of supplyand 2% margin of supply voltage.

EDL Capacitor Power Supply

Returning to FIGS. 2 and 3, EDL capacitor 210 or backup battery 310connected to VBACK 171 are located external to module 100 because theyare physically large and sensitive to temperature. The long term life ofthe EDL capacitors and batteries are sensitive to the ambienttemperature as well as the operating voltage of the capacitor. For thisdesign, the operating voltage has been chosen such that the capacitorwill tolerant ambient temperatures less than 50° C. for at least 10years. In general, the backup power is located near an air intake oranother relatively cool location within the chassis. The backupcontroller 220 performs periodic state-of-health checks on the backuppower source to determine if the power supply is no longer capable ofsustaining and reporting the status through the NVDIMM I2C interface.

Long term lifetime of EDL capacitors show a correlation to temperatureand operating voltage. Like aluminum capacitors, the lifetime generallydoubles for every 10° C. decrease in temperature. Also like aluminumcapacitors, the capacitor is exponentially sensitive to working voltage.Maxwell Technologies models the lifetime of the PC10 capacitors in hoursusing a thermal-nonthermal (T-NT) model:

${L\left( {V,T} \right)} = \frac{{4.8901E} - 06}{V^{7.9838}{\exp\left( {- \frac{9385.8}{T}} \right)}}$where T is the temperature in Kelvin and V is the working voltage involts. This model assumes that at the end of capacitor's lifetime, thecapacitance has decreased 20% from its initial value. A number ofdifferent operating environments are presented in the table below toshow the expected lifetime of the PC10 capacitors:

Temp Working Lifetime Description ° C. Votlage Years Room Temperature 252.50 18 Ambient (high voltage) 40 2.50 3.9 Ambient (reduce voltage) 402.20 11 Operating (high voltage) 50 2.50 1.6 Operating (reduce voltage)50 1.95 11 Server (high voltage) 60 2.50 0.7 Server (reduced voltage) 601.75 11 Server (high voltage) 70 2.50 0.3 Server (reduced voltage) 701.60 10

As shown, the capacitors are operated at low working voltages, whichaffects the structure of the voltage regulator. In a parallelconfiguration, the total capacitance is the sum of all the capacitors.However, the discharge current is large over a small voltage swingduring use. The voltage regulator requires a boost switch-mode powersupply architecture with high-current inductors. In a seriesconfiguration, the total capacitance is the reciprocal of the sum of thereciprocal of the capacitance, but the total working voltage hasincreased. Issues include balancing the operating voltage betweencapacitances and keeping the number of capacitors reasonable. For thepurpose of the design exploration, the 50° C. operating temperature hasbeen chosen allowing the design to use 75% of the capacitor workingvoltage for a 10 year life time. This scenario allows for a 15° C. risein temperature over the common 35° C. external ambient temperature forenterprise computers within a server room. The other target environmentwould be a telecom NEBS standard with a 40° C. maximum ambienttemperature that may increase to 50° C. ambient temperature with a 5° C.higher temperature within the equipment frame during short term HVACfailures. The length of the short-term temperature failures are definedto be up to 96 hours each, but not more than 15 days per year.

FIG. 9 shows an external EDL capacitor backup power supply architecture,which is a more detailed version of FIG. 2. Some embodiments providecapacitor chargers, for example, a capacitor bank charger thatimplements a constant-current, constant-voltage design. The chargerapplies a constant current to the capacitor bank until the bank reachesits final full-charge voltage. At that point, the charger applies aconstant-voltage to float the capacitor bank. The float voltage isapplied because the EDL capacitors have a fairly large leakage currentthat require the balance resistors within the capacitor bank to bebiased to ensure all the capacitors within the bank have equal chargevoltage. The float voltage is programmable by a resistor and accurate to1% as the capacitor bank size may be optimized for each configuration.The float voltage can be set from VCHRG (minus some headroom) to 6volts. The design can leverage LiOn battery charger technology (commonlysingle-ended primary inductive converter or SEPIC architecture), butother techniques can be used. Some chargers require a small processor tomonitor the charge cycle and switch the charger from constant-currentoperation to constant-voltage operation. For these designs, theprocessor within controller 110 can be used depending on the complexityof the algorithm and the hardware connectivity with the charger design.The tables below provide information related to VCHRG and VCAP.

Nominal Supply Name Voltage Accuracy Notes VCHRG   12 V +/−5% 1 VCAP11.5 V +/−1% 2 Notes: 1. Can depend the system, for example, in someembodiments a wider supply range may have an advantage. 2. Nominal floatvoltage of the capacitor bank. During discharge cycle, regulatorscontinue to operate until the capacitor bank discharges to 2.8 V orlower.

Supply Name Min Current Max Current Notes VCHRG  500 mA 1 VCAP 100 mA2700 mA 2 Notes: 1. Based on an example customer specification that thecharger consumes no more than max specification in all cases. 2. Basedon U1 operating at 70% efficiency, U3 operating at 80% efficiency and U4operating at 90% efficiency, that VCAP is operating at 2.8 V (low end ofoperating range) with a 2% loss

The described embodiment also includes a state of health monitor for thebackup power supply. EDL capacitors have a limited life time that issensitive to the working voltage, ambient or storage temperature and thenumber of charge/discharge cycles (wearing). In some applications, onlythe working voltage and the ambient temperature are important, forexample, if the number of predicted cycles is 100 times less thanspecification. The life-time of EDL capacitors is based on thecapacitance degrading to a specified threshold (in most cases 30% dropfrom initial capacitance) or the ESR of the capacitance increasing to aspecified threshold (depending on the manufacturer a 30% to 100%increase from initial ESR). Given the sensitivity of the EDL capacitorto stress, the controller monitors the state-of-health of the capacitor.The state-of-health monitor U11 (of FIG. 9) can be combined with thecharger U6 depending on the implementation. A state of health monitorinforms the FPGA if the capacitor bank is charged sufficiently to handlea loss-of-power. In general, the charger U6 must be able to “turn off”during the test. In, for example, embodiments with a number ofcapacitors in series, the voltage across each capacitor can be monitoredand fed into signals that can be checked over the backup power supplyI2C bus. This allows for the identification of a specific capacitor thathas failed as well as indication that the backup power source has failedoverall.

Measuring the time from power-on (VCHRG power is applied) to when VCAPreaches the fully-charged state provides a method of estimating thehealth of the capacitor bank. Controller 110 can report whether module100 is capable of handling a power loss event. If the capacitor banknever achieves a full-charge state, the system detects this and declaresan error.

To measure capacitance, the capacitor is fully charged. The charger isfirst turned off and a fixed known load (resistor) is applied to thecapacitor bank for a period of time to slightly discharge the capacitor.In general, the load current is small to prevent ESR from affecting themeasurement. The measurement method can be as simple as a voltagecomparator that triggers an interrupt on controller 110 if the VCAPsupply drops below a fixed voltage. If the interrupt is triggered duringthe test, the capacitance is too low and the capacitor bank has failedthe test. One issue is that the capacitor is discharged partially whichmust be accounted for in the energy budget as a power loss event couldoccur right after the self-test.

In order to minimize the cost of the EDL capacitor power supply, theself-test intelligence is located on module 110. To control the logic inthe power supply, an I2C to GPIO expander device is used. Thus,controller 110 is able to control and monitor signals on the backuppower supply module (e.g., 200 or 300).

As discussed, in some cases, a battery may be selected over an EDLcapacitor because batteries have a higher energy density than EDLcapacitors and thus require less volume and mass. For instance, a singleA123 battery is rated for 2.3 Ah at 3.3V, weighs 70 grams and requires 2cubic inches. If module 100 requires 5 Watts for 2 minutes, the requiredenergy is only 0.05 Ah which is over an order of magnitude less than thebattery capacity. Most battery chargers for portable laptop computershave all the necessary functions required for the backup power supply.In addition, most of these devices have an integrated I2C interface formonitoring, configuration and control that can be used by module 100.

SD Data Format

In single SD/MMC+ card operation, the first block of the SD card (bytes0 to 511) is used for SPD and flash configuration purposes. Theremaining blocks in the SD card are used for backup data. During backupoperation, the backup data is read from the first DRAM device in acontinuous byte stream and written into the single flash memory. Thebackup controller then repeats the process for the other DRAM devicesuntil backups of all of the other devices are completed. During restoreoperation, the data is read from the single flash memory in a continuousbyte stream and written into the first DRAM device. The backupcontroller then repeats the process for the other DRAM devices until allthe other devices are completed. The backup controller streams read andwrite data to the flash memory using one single sequential read/writeoperation. This mechanism allows the SD card to perform at maximumbandwidth, but has the side effect that the alignment of each DRAMbackup image may cross SD card block boundaries if the length of eachDRAM backup image is not a multiple of 512 bytes.

For dual SD/MMC+ card operation, the first block of the SD card (bytes 0to 511) in slot 0 is used for SPD and flash configuration purposes. Thefirst block of the other SD card in slot 1 is not used and is ignored.During backup operation, data is read from the first DRAM device in acontinuous byte stream and written to both flash memories at the sametime. The data stream is split into two flash write data streams bysending all even order bytes to slot 0 and all odd order byte to slot 1.The backup controller then repeats the process for the other devicesuntil all the DRAM devices are completed. During restore operation, datais read from both flash memories which is combined by byte interleavingthe data streams to form a single data stream that is written to thefirst DRAM. The backup controller then repeats the process for the otherDRAM devices until all the devices are completed. The backup controllerstreams read and write data to the flash memory using one singlesequential read/write operation for each SD card. This mechanism allowsthe SD cards to perform at maximum bandwidth, but has the side effectthat the alignment of each DRAM backup image may cross SD card blockboundaries if the length of each DRAM backup image is not a multiple of1 kbytes.

For quad SD/MMC+ card operation, the first block of the SD card (bytes 0to 511) in slot 0 is used for SPD and flash configuration purposes. Thefirst block of the other SD card in slot 1 is not used and is ignored.During backup operation, data is read from the first DRAM device in acontinuous byte stream and written to all flash memories at the sametime. The data stream is split into four flash write data streams bysending every 4 bytes to an interface. The backup controller thenrepeats the process for the other devices until all the DRAM devices arecompleted. During restore operation, data is read from both flashmemories which is combined by byte interleaving the data streams to forma single data stream that is written to the first DRAM. The backupcontroller then repeats the process for the other DRAM devices until allthe devices are completed. The backup controller streams read and writedata to the flash memory using one single sequential read/writeoperation for each SD card. This mechanism allows the SD cards toperform at maximum bandwidth, but has the side effect that the alignmentof each DRAM backup image may cross SD card block boundaries if thelength of each DRAM backup image is not a multiple of 2 kbytes.

The table below provides example backup times calculated using a worstcase write/read bandwidth of 20 Mbyte/sec for each SD/MMC card and 40Mbyte/sec for each MMC+ card. The calculation also includes the worstcase SD/MMC+ write interval for updating the flash configuration.

Two Four SD/MMC MMC + Interface Interface One SD/MMC Active ActiveNVDIMM Total Interface Active (40 Mbyte/ (160 Size Data (20 Mbyte/sec)sec) Mbyte/sec) 256 Mbyte  288 Mbyte 15 sec  8 sec 2 sec 512 Mbyte  576Mbyte 30 sec 15 sec 4 sec  1 Gbyte 1152 Mbyte 58 sec 30 sec 8 sec  2Gbyte 2304 Mbyte 116 sec  59 sec 16 sec 

Burn-in Self-Test Operation

DIMM 100 also includes self-test functionality. Self-test can betriggered using the PRODTEST input on the FPGA as well as through the NVI2C interface. The results of the self-test are stored permanently untilthe flash memory is erased through another self-test sequence. In oneexample, self test: takes over the DDR interface (FET switches are off);sets the SELFTEST test in progress bit high; fills DRAM memory with0xA5; fills flash memory with 0x00; turn on progress LED; backups DRAMmemory to flash memory; fill DRAM memory with 0x00; restore DRAM memoryfrom flash memory; tests contents of DRAM memory; and, if an error isdetected sets an error LED and stores the results in flash. If no erroris found, the process loops back to fill the flash memory with 0x00.Self-test methods can be defined by various users of the system of FIG.1, for example, they can be defined by a customer.

Visual Indicators

Visual indications on the board allow for diagnosing system problemswith memory module particularly when multiple modules 100 are usedwithin a system. In the following cases, a slow flash of an LED is 0.25seconds on and 1 second off while a fast flash of an LED is 0.5 secondson and 0.5 seconds off. The memory module has an LED to indicate thebackup operation is occurring. Given that some configurations takemultiple minutes to complete the backup operation, an LED indicates to arepair technician that the module 100 or capacitor bank must not bedisturbed after a power-loss event.

Red LED Description Off System power and capacitor power is off ormemory module is operating in normal operation (POWER UP or IDLE state).If system power is off, the memory module and/or the capacitor bank maybe disconnected. Slow Flash RESTORE operation (flash to DRAM) operationin progress. Fast Flash BACKUP operation (DRAM to flash) operation inprogress. On Restore operation completed, waiting for DRAM to be flushedbefore transiting to IDLE

A visual indication is supplied for the backup power supply to indicatethat the backup power is correctly connected, charging, fully charged orfailed. This can be useful, for example, in a system with multiplemodules 100, it is possible that repair technician must physicallyidentify a failed module or capacitor for replacement.

Green LED Description Off No backup power supply is connected or moduleis open. Slow Flash Backup power supply is charging. Fast Flash Backuppower supply has failed self-test. On Backup power supply is fullycharged.

Visual indication LEDs are also used during burn-in testing. The red LEDlatches on if any of the self-tests failed during the burn-in testing.The green LED will flash during self-test as proof that the self-testoperation is progressing. The green LED toggles at the end of each testperiod (write/read DRAM and write read flash memory) while the testprogresses.

For example, the external system can include various types of systems,for example, a mainframe, a server, a client, a network of varioussystems, etc. Volatile memory 120 can include, for example, Dynamicrandom access memory (DRAM), Z-RAM®, Static random access memory (SRAM),Twin Transistor RAM (TTRAM), etc. Non-volatile memory 130 can include,for example, Read-only memory (ROM), flash memory, Ferroelectric RAM(FeRAM), programmable metallization cell (PMC), etc. In someembodiments, backup power supply 200 can be included as part of DIMM100, while in other embodiments it can be, for example, an externaldevice. Non-volatile memory 130 and volatile memory 120 can be ofvarious sizes and need not be the same size. During a backup operationvarious embodiments can move all data stored in volatile memory 120 tonon-volatile memory 120 or some subset of the data stored in volatilememory 120. The same is true during a restore operation fromnon-volatile memory 120 to non-volatile memory 130. Some embodiments ofFIG. 1 do not include each component and/or function of FIG. 1. Forexample, some embodiments do not include isolation logic 140, someembodiments do store SPD information in volatile memory 130, someembodiments move all data stored in volatile memory 130 to non-volatilememory 120 at the same time (e.g., every DRAM device at once), and someembodiments move the data stored volatile memory 130 to non-volatilememory 120 in chunks, for example, one DRAM device at a time.

Controller 110 can be implemented, for example, using various FPGAs,controllers, processors, and/or memories. In another embodiment,non-volatile controller 110 is an application-specific integratedcircuit (ASIC) that includes a flash chip interface inside thecontroller. By incorporating the flash chip interfaces into the ASICcontroller, external SD/MMC+ controllers are not used, and save/restoreperformance can be improved. In another embodiment, volatile memory 120can be separated into various segments using various staring and endingaddresses. These addresses can be configured by setting registers incontroller 110 through the NVDIMM I2C bus. Which (and in what order) thesegments defined by these addresses should be backuped and/or restoredis also controllable by setting registers in controller 110. Althoughthe invention has been described and illustrated in the foregoingillustrative embodiments, it is understood that the present disclosurehas been made only by way of example, and that numerous changes in thedetails of implementation of the invention can be made without departingfrom the spirit and scope of the invention, which is limited only by theclaims that follow.

What is claimed is:
 1. A memory device comprising: a plurality ofvolatile memory devices; non-volatile memory system including anon-volatile memory device comprising a plurality of storage devices,the non-volatile memory system configured to store in the non-volatilememory device parameters describing one of the plurality of volatilememory devices; an interface for connecting to a backup power sourcearranged to temporarily power the one of the plurality of volatilememory devices upon a loss of power from a primary power source; and acontroller in communication with the one of the plurality of volatilememory devices and the non-volatile memory, wherein the controller isprogrammed to move data from the one of the plurality of volatile memorydevices to the same non-volatile memory device that is configured tostore the non-volatile memory device parameters, upon a loss of power ofthe primary power source at the one of the plurality of the volatilememory devices, and cause the remaining volatile memory devices to enterinto a low power mode; wherein the controller is configured to store thedata to the plurality of storage devices by splitting a data stream,which comprises the data, into a plurality of data streams.
 2. Thememory device of claim 1, wherein the parameters describing the one ofthe plurality of volatile memory devices comprise serial presence detectinformation, a size of the one of the plurality of volatile memorydevices, or a speed of the one of the plurality of volatile memorydevices.
 3. The memory device of claim 1, wherein the controller furthercomprises a cache for storing the parameters.
 4. The memory device ofclaim 1, further comprising a configuration data bus for accessing theparameters.
 5. The memory device of claim 1, further comprising aconfiguration data bus for accessing the parameters, wherein thecontroller further comprises a cache for storing the parameters, andwherein the controller is further programmed to read the parameters fromthe cache in response to receiving a request for the parameters over theconfiguration data bus.
 6. The memory device of claim 1, wherein the oneof the plurality of volatile memory devices comprises a dynamic randomaccess memory.
 7. The memory device of claim 1, wherein the non-volatilememory device comprises electrically erasable programmable read-onlymemory.
 8. The memory device of claim 1, wherein the controllercomprises at least one of an application-specific integrated circuit(ASIC) and a field programmable gate array (FPGA).
 9. The memory deviceof claim 1, wherein the controller is further programmed to move datafrom the non-volatile memory device to the one of the plurality ofvolatile memory devices upon a restoration of power of the primary powersource.
 10. The memory device of claim 1, wherein power from the backuppower source is primarily from a capacitor.
 11. The memory device ofclaim 1, wherein power from the backup power source is primarily from acapacitor.
 12. The memory device of claim 11, wherein the controller isconfigured to estimate a health of the capacitor by measuring an amountof time it takes to fully charge the capacitor.
 13. The memory device ofclaim 1, wherein the data moved by the controller comprises directoryinformation for determining a location of information.
 14. The memorydevice of claim 1, wherein when the power of the primary power sourcereturns before the controller finishes moving the data to thenon-volatile memory device, the controller is further configured toenable retrieval of the data from the one of the plurality of volatilememory devices.
 15. The memory device of claim 1, wherein, upon a lossof power of the primary power source for the plurality of volatilememory devices, the controller is programmed to move data from each ofthe plurality of volatile memory devices to the same non-volatile memorydevice in a sequential manner.
 16. A method, comprising: upon a loss ofpower of a primary power source at one of a plurality of volatile memorydevices: causing other volatile memory devices to enter into a low powermode; and moving data from the one of the volatile memory devices to anon-volatile memory system, including a non-volatile memory devicecomprising a plurality of storage devices, based on parametersdescribing the one of the volatile memory devices, while the one of thevolatile memory devices is temporarily powered by a backup power source,wherein the non-volatile memory device is configured to store both thedata moved from the one of the volatile memory devices and theparameters describing the one of the volatile memory; wherein moving thedata comprises storing the data to the plurality of storage devices bysplitting a data stream, which comprises the data, into a plurality ofdata streams.
 17. The method of claim 16, wherein the parameterscomprise serial presence detect information, a size of the one of thevolatile memory devices, or a speed of the one of the volatile memorydevices.
 18. The method of claim 16, further comprising moving data fromthe non-volatile memory device to the one of the volatile memory devicesupon a restoration of power of the primary power source.
 19. The methodof claim 16, wherein power from the backup power source is primarilyfrom a capacitor.